# ^8 Q' M& u; y5 A1 Q来!不说那些没用的。整点实际的给大家。对了!完整代码仅供学习参考使用,有需要可以联系作者获取 5 C. m+ n5 T" R6 M9 ]& F% V p. ~/ {& v( m8 W6 W
首先打开她给我发的链接企名片,在未登陆的情况下看到网站确实是有一些让爬虫上瘾的数据。嘿!你左手右手一个慢动作,跟着我的操作一起走呀!光看着也不抵用噻 , l" p2 |( J& T. L: f8 z. d- Y: b% W) `2 N+ @( x
刷新页面,开发者工具打开,可以发现网页是js动态生成的,并没有发现你咱们想要的HTML。象征性的点击一下查看响应数据 4 e- p6 Q* s/ V. n 2 F& C3 s, R) J& Z9 F3 J 我不知道productListVip这个接口内的响应数据是不是就有网站我们需要的数据!但是可以看到有一个数encrypt_data是经过处理的,初步看这只是一个Base64的编码,真的这么简单吗?我想它在编码之前是经过加密算法处理过的。因为很多网站对数据加密或多或少都是这样的套路,所以为了验证这个被处理的东东到底是个啥,我们需要先去破解它 % [. u. ^* e( |5 e9 h u$ I3 T 7 j' s3 C' w. Y% n1 f. T6 |: n这个过程是快乐的,当然!它建立在你成功或有收获的基础之上。不然用Selenium来做它难道不香吗?还要什么自行车?, x2 s' e( q2 i$ s7 k* u4 a# i# b
参数encrypt_data既然是经过处理的,但是终归是要给用户看的,所以在它呈现到网页过程中肯定是有在中间的某个地方对这个数据做了对应的解密操作,不然搞一顿Base64的编码数据给用户看?毕竟大家不是程序员呀! w2 t* }1 C3 Z( }$ G W( R* `- w; R: n c6 ?4 z
之前说到过对于一些请求、响应…参数加密的这种场景,去反推的思路有很多种,可以通过搜索关键字、也可以通过hook、也可以断点。思路是这样的,具体因站而异哈 % a. W3 K1 i* ~$ S$ \8 {. ?; P6 S+ E# L: ~, J
既然是JS生成我们在开发者工具中点击Sources,因为现在加密参数的那个请求它就是XHR,所以在右侧XHR断点调试栏添加它,这个功能支持XHR请求断点调试!' ?4 g! o& y9 y+ T
& l: i B/ T/ n( r1 E
OK!做好上面的一切我们刷新网站页面。然后断点跟预期中的一样,停了下来。我们单步跟进可以很直观的看到每一个参数传递的细节,直到发现了跟接口响应数据极度相似的参数e.data7 M1 ]# n8 p+ y
& V) M) h" _5 I5 P0 j8 f& ^" F继续跟进发现e.data参数的数据通过一个函数实现。可以在Console中将这个函数调用执行一下。没错!函数Obiect(u.a)(e.encrypt_data)返回的内容正是网站页面呈现的数据。到了这一步我们能确定接口就是数据加载的接口,Base64编码也正是经过加密之后的数据$ q; U ?9 ~: Q, s0 K' c
4 U/ f% J$ l8 [/ o1 t \既然现在我们已经找到了真实的数据,那就证明它在呈现的时候就已经解密了,我们进到这个函数体内去继续寻找,发现了解密函数 6 ?! M3 n2 J! }& D3 {# u& A , P! s& E) C% v* z- D: M" wfunction o(t)是一个解密函数,可以看到它先调用s函数,s函数是什么?等下我们进去看看就知道了!一共有四个参数,a.a.decode(t)这个参数是数据。等下下面看了s函数就明白了。另外三个参数固定的,最后JSON.parse格式化json数据 5 I. o( E s1 O5 L5 h" y& I q( j! p" a0 X
这个就是s函数,你觉得它主要功能是干嘛的呢?我猜它是做数据解密的,你看看这一大串不是解密算法就让我写代码不掉头发!这东西你要想了解它解密算法的原理用Python实现解密的话可以好好分析一下4 U+ s' [$ D7 M
function s(t, e, i, n, a, s) { k g5 Q4 d7 ~- z8 e+ [; ]% u
var o, r, c, l, u, d, h, p, f, v, m, g, b, y, _ = new Array(16843776,0,65536,16843780,16842756,66564,4,65536,1024,16843776,16843780,1024,16778244,16842756,16777216,4,1028,16778240,16778240,66560,66560,16842752,16842752,16778244,65540,16777220,16777220,65540,0,1028,66564,16777216,65536,16843780,4,16842752,16843776,16777216,16777216,1024,16842756,65536,66560,16777220,1024,4,16778244,66564,16843780,65540,16842752,16778244,16777220,1028,66564,16843776,1028,16778240,16778240,0,65540,66560,0,16842756), C = new Array(-2146402272,-2147450880,32768,1081376,1048576,32,-2146435040,-2147450848,-2147483616,-2146402272,-2146402304,-2147483648,-2147450880,1048576,32,-2146435040,1081344,1048608,-2147450848,0,-2147483648,32768,1081376,-2146435072,1048608,-2147483616,0,1081344,32800,-2146402304,-2146435072,32800,0,1081376,-2146435040,1048576,-2147450848,-2146435072,-2146402304,32768,-2146435072,-2147450880,32,-2146402272,1081376,32,32768,-2147483648,32800,-2146402304,1048576,-2147483616,1048608,-2147450848,-2147483616,1048608,1081344,0,-2147450880,32800,-2147483648,-2146435040,-2146402272,1081344), w = new Array(520,134349312,0,134348808,134218240,0,131592,134218240,131080,134217736,134217736,131072,134349320,131080,134348800,520,134217728,8,134349312,512,131584,134348800,134348808,131592,134218248,131584,131072,134218248,8,134349320,512,134217728,134349312,134217728,131080,520,131072,134349312,134218240,0,512,131080,134349320,134218240,134217736,512,0,134348808,134218248,131072,134217728,134349320,8,131592,131584,134217736,134348800,134218248,520,134348800,131592,8,134348808,131584), x = new Array(8396801,8321,8321,128,8396928,8388737,8388609,8193,0,8396800,8396800,8396929,129,0,8388736,8388609,1,8192,8388608,8396801,128,8388608,8193,8320,8388737,1,8320,8388736,8192,8396928,8396929,129,8388736,8388609,8396800,8396929,129,0,0,8396800,8320,8388736,8388737,1,8396801,8321,8321,128,8396929,129,1,8192,8388609,8193,8396928,8388737,8193,8320,8388608,8396801,128,8388608,8192,8396928), k = new Array(256,34078976,34078720,1107296512,524288,256,1073741824,34078720,1074266368,524288,33554688,1074266368,1107296512,1107820544,524544,1073741824,33554432,1074266112,1074266112,0,1073742080,1107820800,1107820800,33554688,1107820544,1073742080,0,1107296256,34078976,33554432,1107296256,524544,524288,1107296512,256,33554432,1073741824,34078720,1107296512,1074266368,33554688,1073741824,1107820544,34078976,1074266368,256,33554432,1107820544,1107820800,524544,1107296256,1107820800,34078720,0,1074266112,1107296256,524544,33554688,1073742080,524288,0,1074266112,34078976,1073742080), T = new Array(536870928,541065216,16384,541081616,541065216,16,541081616,4194304,536887296,4210704,4194304,536870928,4194320,536887296,536870912,16400,0,4194320,536887312,16384,4210688,536887312,16,541065232,541065232,0,4210704,541081600,16400,4210688,541081600,536870912,536887296,16,541065232,4210688,541081616,4194304,16400,536870928,4194304,536887296,536870912,16400,536870928,541081616,4210688,541065216,4210704,541081600,0,541065232,16,16384,541065216,4210704,16384,4194320,536887312,0,541081600,536870912,4194320,536887312), A = new Array(2097152,69206018,67110914,0,2048,67110914,2099202,69208064,69208066,2097152,0,67108866,2,67108864,69206018,2050,67110912,2099202,2097154,67110912,67108866,69206016,69208064,2097154,69206016,2048,2050,69208066,2099200,2,67108864,2099200,67108864,2099200,2097152,67110914,67110914,69206018,69206018,2,2097154,67108864,67110912,2097152,69208064,2050,2099202,69208064,2050,67108866,69208066,69206016,2099200,0,2,69208066,0,2099202,69206016,2048,67108866,67110912,2048,2097154), L = new Array(268439616,4096,262144,268701760,268435456,268439616,64,268435456,262208,268697600,268701760,266240,268701696,266304,4096,64,268697600,268435520,268439552,4160,266240,262208,268697664,268701696,4160,0,0,268697664,268435520,268439552,266304,262144,266304,262144,268701696,4096,64,268697664,4096,266304,268439552,64,268435520,268697600,268697664,268435456,262144,268439616,0,268701760,262208,268435520,268697600,268439552,268439616,0,268701760,266240,266240,4160,4160,262208,268435456,268701696), S = function(t) { - g, s+ T& |6 P7 p! |0 u# s; j for (var e, i, n, a = new Array(0,4,536870912,536870916,65536,65540,536936448,536936452,512,516,536871424,536871428,66048,66052,536936960,536936964), s = new Array(0,1,1048576,1048577,67108864,67108865,68157440,68157441,256,257,1048832,1048833,67109120,67109121,68157696,68157697), o = new Array(0,8,2048,2056,16777216,16777224,16779264,16779272,0,8,2048,2056,16777216,16777224,16779264,16779272), r = new Array(0,2097152,134217728,136314880,8192,2105344,134225920,136323072,131072,2228224,134348800,136445952,139264,2236416,134356992,136454144), c = new Array(0,262144,16,262160,0,262144,16,262160,4096,266240,4112,266256,4096,266240,4112,266256), l = new Array(0,1024,32,1056,0,1024,32,1056,33554432,33555456,33554464,33555488,33554432,33555456,33554464,33555488), u = new Array(0,268435456,524288,268959744,2,268435458,524290,268959746,0,268435456,524288,268959744,2,268435458,524290,268959746), d = new Array(0,65536,2048,67584,536870912,536936448,536872960,536938496,131072,196608,133120,198656,537001984,537067520,537004032,537069568), h = new Array(0,262144,0,262144,2,262146,2,262146,33554432,33816576,33554432,33816576,33554434,33816578,33554434,33816578), p = new Array(0,268435456,8,268435464,0,268435456,8,268435464,1024,268436480,1032,268436488,1024,268436480,1032,268436488), f = new Array(0,32,0,32,1048576,1048608,1048576,1048608,8192,8224,8192,8224,1056768,1056800,1056768,1056800), v = new Array(0,16777216,512,16777728,2097152,18874368,2097664,18874880,67108864,83886080,67109376,83886592,69206016,85983232,69206528,85983744), m = new Array(0,4096,134217728,134221824,524288,528384,134742016,134746112,16,4112,134217744,134221840,524304,528400,134742032,134746128), g = new Array(0,4,256,260,0,4,256,260,1,5,257,261,1,5,257,261), b = t.length > 8 ? 3 : 1, y = new Array(32 * b), _ = new Array(0,0,1,1,1,1,1,1,0,1,1,1,1,1,1,0), C = 0, w = 0, x = 0; x < b; x++) { 2 d0 X( ~$ u; q+ O; e; Z var k = t.charCodeAt(C++) << 24 | t.charCodeAt(C++) << 16 | t.charCodeAt(C++) << 8 | t.charCodeAt(C++) 9 O7 o9 \$ ]( @, L , T = t.charCodeAt(C++) << 24 | t.charCodeAt(C++) << 16 | t.charCodeAt(C++) << 8 | t.charCodeAt(C++); $ @. z4 i; @4 Z! {, q. u/ O k ^= (n = 252645135 & (k >>> 4 ^ T)) << 4,% Y' B" C/ B5 J. o6 P; W F. a
k ^= n = 65535 & ((T ^= n) >>> -16 ^ k),1 ]: x% \* d* X4 B
k ^= (n = 858993459 & (k >>> 2 ^ (T ^= n << -16))) << 2, - `- o4 ]* S! P F( i1 Z5 e [ k ^= n = 65535 & ((T ^= n) >>> -16 ^ k), 0 b3 f4 k4 o: n; ~5 n k ^= (n = 1431655765 & (k >>> 1 ^ (T ^= n << -16))) << 1, 7 v( b4 c% W' j A k ^= n = 16711935 & ((T ^= n) >>> 8 ^ k), ) K; r" `$ v7 n# o7 U4 ? n = (k ^= (n = 1431655765 & (k >>> 1 ^ (T ^= n << 8))) << 1) << 8 | (T ^= n) >>> 20 & 240, ! ]; Z. g) u; _& r9 c k = T << 24 | T << 8 & 16711680 | T >>> 8 & 65280 | T >>> 24 & 240,( A0 {; d" \& W* z t+ D, @# G& @
T = n; ( Q4 L9 p0 D* t for (var A = 0; A < _.length; A++)4 G, v* `$ v6 C( ~, M" G9 v
_[A] ? (k = k << 2 | k >>> 26,3 j9 x, s1 b3 T+ l9 F
T = T << 2 | T >>> 26) : (k = k << 1 | k >>> 27,$ S( ] Z" K( j: R* U
T = T << 1 | T >>> 27), $ ]9 V+ s* y4 h. ^' _1 \ T &= -15, h F2 p$ J4 D/ g" e% h
e = a[(k &= -15) >>> 28] | s[k >>> 24 & 15] | o[k >>> 20 & 15] | r[k >>> 16 & 15] | c[k >>> 12 & 15] | l[k >>> 8 & 15] | u[k >>> 4 & 15], " q# ` Q/ C) a! b! C+ r i = d[T >>> 28] | h[T >>> 24 & 15] | p[T >>> 20 & 15] | f[T >>> 16 & 15] | v[T >>> 12 & 15] | m[T >>> 8 & 15] | g[T >>> 4 & 15], # L% A% a$ e6 I& r0 D5 }/ I j n = 65535 & (i >>> 16 ^ e), 3 I9 h, t3 V& L! a y[w++] = e ^ n, : U8 f7 n( ?1 Z; B5 D/ i( j y[w++] = i ^ n << 167 f( f) e& o$ Q, O$ ^( t
} ) C5 F6 M' N( k0 h2 u$ ] return y ) q) Y) M% m" M4 C5 F! { }(t), z = 0, B = e.length, I = 0, j = 32 == S.length ? 3 : 9; 0 b% H5 F9 |0 N6 z p = 3 == j ? i ? new Array(0,32,2) : new Array(30,-2,-2) : i ? new Array(0,32,2,62,30,-2,64,96,2) : new Array(94,62,-2,32,64,2,30,-2,-2), ) ]" |) ~! @. a. |$ E2 R 2 == s ? e += " " : 1 == s ? i && (c = 8 - B % 8,* q6 t4 q" i$ S% t& V
e += String.fromCharCode(c, c, c, c, c, c, c, c),$ d, }3 W, i1 g: u2 F( @
8 === c && (B += 8)) : s || (e += "\0\0\0\0\0\0\0\0"); * f9 c' F7 B# M3 g5 h! E var F = "" 2 `* T8 U" h6 A9 T( V, [0 ^9 x , $ = "";3 @+ g' {$ o$ ^ Y8 U: I2 a
for (1 == n && (f = a.charCodeAt(z++) << 24 | a.charCodeAt(z++) << 16 | a.charCodeAt(z++) << 8 | a.charCodeAt(z++),* T5 t7 c& ]) ?( C5 O
m = a.charCodeAt(z++) << 24 | a.charCodeAt(z++) << 16 | a.charCodeAt(z++) << 8 | a.charCodeAt(z++),2 l$ q1 p2 A( t9 o: t' P
z = 0); z < B; ) { 2 W0 ]& ` n0 s2 f" v. v for (d = e.charCodeAt(z++) << 24 | e.charCodeAt(z++) << 16 | e.charCodeAt(z++) << 8 | e.charCodeAt(z++), / z; ^& ~9 {8 S: r1 w h = e.charCodeAt(z++) << 24 | e.charCodeAt(z++) << 16 | e.charCodeAt(z++) << 8 | e.charCodeAt(z++), ! x' Q6 A6 y* \+ `; ^ 1 == n && (i ? (d ^= f,, t$ z8 V, R3 V4 r
h ^= m) : (v = f, . r5 ]- \- K; G g = m, " {: H1 z8 z* P" o. Z2 Z f = d,7 Z" p0 I; c5 B( m' S+ j( B
m = h)),2 W4 ?7 c, ?; F+ s3 U
d ^= (c = 252645135 & (d >>> 4 ^ h)) << 4,. S3 y$ G/ F" X' s
d ^= (c = 65535 & (d >>> 16 ^ (h ^= c))) << 16,) j. J* R. c2 `. ~& b
d ^= c = 858993459 & ((h ^= c) >>> 2 ^ d),; g# B5 |/ X; }7 ~7 j- c4 D+ O
d ^= c = 16711935 & ((h ^= c << 2) >>> 8 ^ d), ( [* u0 t/ F7 p! w1 f d = (d ^= (c = 1431655765 & (d >>> 1 ^ (h ^= c << 8))) << 1) << 1 | d >>> 31, : A) ?+ _* G7 }. j9 N# Z h = (h ^= c) << 1 | h >>> 31, / x, O, L8 f# b" Z) U r = 0; r < j; r += 3) {2 Q g: \4 k% x
for (b = p[r + 1], 7 _: @" \ I) s- K6 l y = p[r + 2], e0 o8 ?6 G5 K3 s9 P4 b- n6 V
o = p[r]; o != b; o += y) ; Q" B* r) x" {5 C. K5 E l = h ^ S[o],' W5 ]/ W6 f4 ?- d. s2 ]
u = (h >>> 4 | h << 28) ^ S[o + 1],' B0 _0 I D" Y, @8 o! r6 A
c = d, 4 ]4 S! ?. A' q$ Q9 k d = h,/ Q9 v G9 K6 h& V" ^4 q7 E
h = c ^ (C[l >>> 24 & 63] | x[l >>> 16 & 63] | T[l >>> 8 & 63] | L[63 & l] | _[u >>> 24 & 63] | w[u >>> 16 & 63] | k[u >>> 8 & 63] | A[63 & u]); 4 ~8 I M$ n2 G M c = d,9 v/ P* T0 t7 m9 u( L
d = h,8 s, J0 R) I% h6 Q7 x
h = c % Q8 A2 Y# E. h7 y } 3 x* A5 t I6 L2 a- b h = h >>> 1 | h << 31,4 u6 r; [) ]' ] ^% ]( {
h ^= c = 1431655765 & ((d = d >>> 1 | d << 31) >>> 1 ^ h),+ T5 ^) ^7 i- d0 I' d" i
h ^= (c = 16711935 & (h >>> 8 ^ (d ^= c << 1))) << 8, 1 r7 p* o% j, f% J2 a+ L- \+ X h ^= (c = 858993459 & (h >>> 2 ^ (d ^= c))) << 2, $ s% v) \ k8 O$ u& J. O4 Q h ^= c = 65535 & ((d ^= c) >>> 16 ^ h),3 b* M) K9 e* R0 k4 y
h ^= c = 252645135 & ((d ^= c << 16) >>> 4 ^ h),3 Y( ? |8 T2 w3 } S3 L
d ^= c << 4,0 h4 Y# H! D2 J5 u9 U* B8 f
1 == n && (i ? (f = d,3 F- P5 B5 Z. p
m = h) : (d ^= v, 9 s- N( f c+ ]; _. {. W: d h ^= g)), * X7 W2 u: M7 B. c! _ $ += String.fromCharCode(d >>> 24, d >>> 16 & 255, d >>> 8 & 255, 255 & d, h >>> 24, h >>> 16 & 255, h >>> 8 & 255, 255 & h), # R" y; a. q& B4 o. W 512 == (I += 8) && (F += $, k6 D2 F2 R* y3 D& P5 J) X $ = "", 8 z2 E# y/ m* |% M+ {6 { I = 0) . S+ z6 P% S' V. M }! r& M1 F" H z
if (F = (F += $).replace(/\0*$/g, ""), $ x: k/ R# v) L7 }! @) @ !i) {) D6 M2 V: M' `0 c" \# e
if (1 === s) {; v! Z4 T5 B4 k) U) H T6 J3 O
var N = 0;" X) C5 z s! I$ P+ M) |7 Y/ i
(B = F.length) && (N = F.charCodeAt(B - 1)), ' A; W* ]$ g3 K0 O N <= 8 && (F = F.substring(0, B - N)) + O. v9 y, c! f- R! b* d" i } + R9 n) B: g7 O2 U F = decodeURIComponent(escape(F))" M/ \0 U. b* X
}) O# x3 ]$ L; V1 J
return F3 J- b: f$ j, p6 G5 Y5 I
} - _% w2 ]6 h; l: L* n8 R. j4 b! v. m2 h/ L- H2 }, k* O