/* remove HTTP headers from multiple gzip or single zip from stdin */

    int fileno(FILE *); 
    #define jmp (yy_start) = 1 + 2 *
    #define echo do{if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0)

   xa "\x1f\x8b"    
   /* GZIP magic number */
   xb "\x50\x4B\x03\x04"
   /* ZIP magic number */

   %s xa
   %option noyywrap noinput nounput
   %%
   {xa}|{xb} echo;jmp xa;
   "HTTP" jmp 0;
   .|\n echo;
   \n|.
   %%
   int main(){ yylex();exit(0);}

Comments URL: https://news.ycombinator.com/item?id=34243659

Points: 2

# Comments: 0

   /* chunked transfer decoding */
   
    #define echo do{if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0)
    #define jmp (yy_start) = 1 + 2 *
    int fileno (FILE *);
    int ischunked,chunksize,count;
   xa "\15"|"\12"
   xb "\15\12" 
   xc "HTTP/0.9"|"HTTP/1.0"|"HTTP/1.1"
   xd [Cc][Hh][Uu][Nn][Kk][Ee][Dd]
   xe [0-9a-fA-F]+\r\n
   xf [0-9a-fA-F]*\r\n
   %option noyywrap nounput noinput 
   %s xb xc xd xe xf
   %%
   ^{xc} echo;ischunked=0;jmp xc;
   ^transfer-encoding: echo;jmp xb;
   \r\n\r\n echo;jmp xe;
   {xd} echo;ischunked=1;
   {xf}|{xe} {
   count=0;
   if(ischunked==1)
   {chunksize=strtol(yytext,NULL,16);
   jmp xd;};
   };
   {xb} jmp xf;
   . { 
   count++;
   if(count==chunksize)jmp xe;
   echo;
   };
   ^[A-Fa-f0-9]+{xa}
   {xa}+[A-Fa-f0-9]+{xa}
   {xb}[A-Fa-f0-9]+{xb}
   %%
   int main(){ yylex();exit(0);}

Points: 1

# Comments: 1

Solution:

       #!/bin/sh
       # usage: $0 pattern < file
       x=$(echo x|tr x '\34');
       tr -d '\12'|sed "s/$x//g;s/$1/$x&$x/g"|tr $x '\12'|sed -n "/^$1/p"

Comments URL: https://news.ycombinator.com/item?id=30186762

Points: 2

# Comments: 1

The script uses a feature of HTTP/1.1 called pipelining; proponents of HTTP/2 and HTTP/3 want people to believe it has problems because it does not fit their commercialised web business model. As demonstrated by the script below, it has no problems. It's a feature that simply does not suit the online ad industry-funded business model with its gigantic corporate browser, bloated conglomeration web pages and incessant data collection. Here, only 2 TCP connections are used to retrieve 141 images. Most servers are less restrictive and allow more than 100 requests per TCP connection. Pipelining works great. Much more efficient than browsers which open hundreds of connections. IMHO.

    (export Connection=keep-alive
    x1=http://www.minimizedistraction.com/img/vrg_google_doc_final_vrs03-
    x2(){ seq -f "$x1%g.jpg" $1 $2;};
    x3(){ yy025|nc -vvn 173.236.175.199 80;};
    x2   1 100|x3;
    x2 101 200|x3;
    )|exec yy056|exec od -An -tx1 -vw99999|exec tr -d '\40'|exec sed 's/ffd9ffd8/ffd9\
    ffd8/g'|exec sed -n /ffd8/p|exec split -l1;
    for x in x??;do xxd -p -r < $x > $x.jpg;rm $x;done;
    convert x??.jpg 1.pdf 2>/dev/null;rm x??.jpg

    ls -l ./1.pdf

Retrieving 100s of small JPEG files with a single TCP connection. Here, 2 TCP connections for 141 files.

Splitting the output into image files and making a PDF from the images.

yy025 can be found here:

https://news.ycombinator.com/item?id=27178484

https://news.ycombinator.com/item?id=27180125

yy056 can be found here:

https://news.ycombinator.com/item?id=27769305

    # std utils: split, seq, od, tr, sed, flex
    # common utils: nc, xxd, imagemagick
    # simple utils made with flex: yy025, yy056

    (export Connection=keep-alive
    x1=http://www.minimizedistraction.com/img/vrg_google_doc_final_vrs03-
    x2(){ seq -f "$x1%g.jpg" $1 $2;};
    x3(){ yy025|nc -vvn 173.236.175.199 80;};
    x2   1 100|x3;
    x2 101 200|x3;
    )|exec yy056|exec od -An -tx1 -vw99999|exec tr -d '\40'|exec sed 's/ffd9ffd8/ffd9\
    ffd8/g'|exec sed -n /ffd8/p|exec split -l1;
    for x in x??;do xxd -p -r < $x > $x.jpg;rm $x;done;
    convert x??.jpg 1.pdf 2>/dev/null;rm x??.jpg

    firefox ./1.pdf

Comments URL: https://news.ycombinator.com/item?id=27769701

Points: 2

# Comments: 0

      cat > 056.l
 
        /* JPEG file carver */
    
     #define echo ECHO
     #define jmp BEGIN
     int fileno(FILE *);
    
    xa "\xFF\xD8"
    xb "\xFF\xD9" 
    
    %s xa 
    %option noyywrap nounput noinput
    %%
    
    {xa} echo;jmp xa;
    {xb} echo;jmp 0;
    .|\n|\r echo;
    .|\n
    
    %%
    int main(){ yylex();exit(0) ;}
    
    ^D

    flex 056.l
    cc -static -o yy056 lex.yy.c

Comments URL: https://news.ycombinator.com/item?id=27769305

Points: 1

# Comments: 0

The "gibberish" is GZIP compressed data. "yy054" is a simple filter I wrote to extract a GZIP file from stdin, i.e., discard leading and trailing garabage. As far as I can tell, the compressed file "ee.txt" is not chunked transfer encoded. If it was chunked we would first extract the GZIP, then decompress and finally process the chunks (e.g., filter out the chunk sizes with the filter submitted in the OP).

In this case all we need to do is extract the GZIP file "ee.txt" from stdin, then decompress it:

    printf "GET /ee.txt\r\nHost: stuff-storage.sfo3.digitaloceanspaces.com\r\nConnection: close\r\n\r\n"|openssl s_client -connect 138.68.34.161:443 -quiet|yy054|gzip -dc > 1.htm
    firefox ./1.htm
   

    cat > 054.l;    
     int fileno(FILE *);
     #define jmp BEGIN
     #define echo ECHO
    
    xa "\x1f\x8b"    
     /* GZIP magic number */
    
    %s xa
    %option noyywrap noinput nounput
    %%
    {xa} echo;jmp xa;
    .|\n echo;
    \n|.
    %%
    int main(){ yylex();exit(0);}
    ^D

    flex -8Crf 054.l
    cc -pipe -static lex.yy.c -o yy054

    example usage:
    printf 'GET /file.gz\r\nHost: example.com\r\nConnection: close\r\n\r\n" |nc -vv example.com 80|yy054|zcat

Comments URL: https://news.ycombinator.com/item?id=27490265

Points: 3

# Comments: 1

The compiled program is only useful for filtering chunked transfer encoding on stdin. Most "HTTP clients" like wget or curl already take care of processing chunked transfer encoding. It is when working with something like netcat that chunked tranfser encoding becomes "DIY". This is a simple program that attempts to solve that problem. It could be written by hand without using flex.

   flex -8iCrfa <

   printf 'GET / HTTP/1.1\r\nHost: us.yahoo.com\r\nConnection: close\r\n\r\n'|openssl s_client -connect us.yahoo.com:443 -ign_eof|./yy045