CURL CURL YLMRX YLMRX Created: 2019-07-03 Wed 11:43 1 ABOUT - - PowerPoint PPT Presentation

▶
curl curl
SMART_READER_LITE
LIVE PREVIEW

CURL CURL YLMRX YLMRX Created: 2019-07-03 Wed 11:43 1 ABOUT - - PowerPoint PPT Presentation

Dec 21, 2022 254 likes •974 views

CURL CURL YLMRX YLMRX Created: 2019-07-03 Wed 11:43 1 ABOUT ABOUT Yoann " fuzzy " Lamouroux: Reverse-engineer and security expert @dataimpact (we're hiring ) Former sysadmin Trol Wdocumented opinions: xoxopowo@twitter

slide-1
SLIDE 1

CURL CURL

YLMRX YLMRX

Created: 2019-07-03 Wed 11:43

1

slide-2
SLIDE 2

ABOUT ABOUT

Yoann "fuzzy" Lamouroux: Reverse-engineer and security expert @dataimpact (we're hiring ) Former sysadmin Trol ˆ Wdocumented opinions: xoxopowo@twitter legreffier@irc.freenode.net

slide-3
SLIDE 3

ABOUT LAST YEAR ABOUT LAST YEAR

slide-4
SLIDE 4

ABOUT LAST YEAR ABOUT LAST YEAR

5' is short (except when prod is down)

slide-5
SLIDE 5

ABOUT LAST YEAR ABOUT LAST YEAR

5' is short (except when prod is down) Now I have 20 (w00t)

slide-6
SLIDE 6

ABOUT LAST YEAR ABOUT LAST YEAR

5' is short (except when prod is down) Now I have 20 (w00t) I hope I deal better with time

slide-7
SLIDE 7

ABOUT LAST YEAR ABOUT LAST YEAR

5' is short (except when prod is down) Now I have 20 (w00t) I hope I deal better with time (so I made a slide about dealing with time)

slide-8
SLIDE 8

ABOUT LAST YEAR ABOUT LAST YEAR

5' is short (except when prod is down) Now I have 20 (w00t) I hope I deal better with time (so I made a slide about dealing with time) No more curling jokes (sorry)

slide-9
SLIDE 9

TRIVIA TRIVIA

Project started in 1996 Still maintained by Daniel Stenberg (@badger) libcurl for about every language out there The curl binary is in EVERY default install

slide-10
SLIDE 10

ALL OF THEM ALL OF THEM

slide-11
SLIDE 11

ALL OF THEM ALL OF THEM

GNU/Linux, *BSD

slide-12
SLIDE 12

ALL OF THEM ALL OF THEM

GNU/Linux, *BSD MacOS

slide-13
SLIDE 13

ALL OF THEM ALL OF THEM

GNU/Linux, *BSD MacOS Windows 10 (recently)

slide-14
SLIDE 14

SOME QUESTIONS SOME QUESTIONS

curl is old curl is badly documented (?) DevTools (Firefox, Chrome) is good httpie is neater/prettier python-requests

slide-15
SLIDE 15

SOME ANSWERS SOME ANSWERS

Old means: Good Stable/reliable DevTools are indeed good httpie is a curl wrapper python-requests is python (hang-on, brb)

slide-16
SLIDE 16

DOCUMENTATION DOCUMENTATION

You usually need curl in critical situations No time to dig through 3k lines manual

slide-17
SLIDE 17

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply:

slide-18
SLIDE 18

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies

slide-19
SLIDE 19

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes

slide-20
SLIDE 20

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes (aka. mai', aka. Kloug{Front,Flare,…})

slide-21
SLIDE 21

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes (aka. mai', aka. Kloug{Front,Flare,…}) … whatever cool kids use these days

slide-22
SLIDE 22

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes (aka. mai', aka. Kloug{Front,Flare,…}) … whatever cool kids use these days And shiny boxes (aka. docker)

slide-23
SLIDE 23

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes (aka. mai', aka. Kloug{Front,Flare,…}) … whatever cool kids use these days And shiny boxes (aka. docker) Tighter firewall policy (aka. no internets)

slide-24
SLIDE 24

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes (aka. mai', aka. Kloug{Front,Flare,…}) … whatever cool kids use these days And shiny boxes (aka. docker) Tighter firewall policy (aka. no internets) Just because you can run Chrome in docker,

slide-25
SLIDE 25

EVERYWHERE EVERYWHERE

DevTools won't get you far beyond the browser Today's IT imply: Reverse-proxies Cloudy jokes (aka. mai', aka. Kloug{Front,Flare,…}) … whatever cool kids use these days And shiny boxes (aka. docker) Tighter firewall policy (aka. no internets) Just because you can run Chrome in docker, … doesn't mean you should

slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30

BASICS BASICS

Display body on stdout.

>> curl https://www.example.com/

slide-31
SLIDE 31

VERBOSE VERBOSE

 curl -v https://httpbin.org > /dev/null * Rebuilt URL to: https://httpbin.org/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 34.230.136.58... * TCP_NODELAY set * Connected to httpbin.org (34.230.136.58) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * (304) (OUT), TLS handshake, Client hello (1): } [512 bytes data] * (304) (IN), TLS handshake, Server hello (2): { [89 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4832 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12):

slide-32
SLIDE 32

PREFIXES: PREFIXES:

* : is information > : protocol verbose FROM your computer (*) < : protocol verbose TO your computer (*) } : encrypted data FROM your computer { : encrypted data TO your computer [xxx] : size (in bytes) of data transferred. (ssl verbose with brackets is shown only when stdout is redirected) (*) : doesn't mean it's not encrypted

slide-33
SLIDE 33

MORE VERBOSE MORE VERBOSE

tcpdump might not be the answer (yet).

  • -trace and --trace-ascii for byte-per-byte analysis.

Use - or filename as an argument to write to stdout or to a file.

slide-34
SLIDE 34

CUSTOM HEADERS CUSTOM HEADERS

  • H (or --header) : to send custom headers

Add 'Key: Value' for each headers

  • A foo: is a shortcut to -H 'User-Agent: foo'
  • b foo=bar: is a shortcut to -H 'Cookie: foo=bar'

(Cookies are just headers your browser is used to save)

slide-35
SLIDE 35

COOKIES COOKIES

Not saved by default Use -c to save cookies to a file (- to display on stdout) Use -b to read from a file (it won't by default)

slide-36
SLIDE 36

TIMER AFTER TIME TIMER AFTER TIME

slide-37
SLIDE 37

Have-you ever seen this ?: time curl http://example.org

slide-38
SLIDE 38

TRY : TRY :

curl --trace-time -v http://example.org (Only works in verbose or trace mode)

slide-39
SLIDE 39

TRY : TRY :

curl --trace-time -v http://example.org (Only works in verbose or trace mode) Unless you do want to check the cpu-time / user-time of an HTTP client request.

slide-40
SLIDE 40

TRY : TRY :

curl --trace-time -v http://example.org (Only works in verbose or trace mode) Unless you do want to check the cpu-time / user-time of an HTTP client request. (you don't)

slide-41
SLIDE 41

ANOTHER APPROACH ANOTHER APPROACH

You can write many variables on output, with the format string

  • ption including:

Request information: http_code http_version Time and speed: time_total speed_download Many more…

slide-42
SLIDE 42

FOR EXAMPLE: FOR EXAMPLE:

Introducing -oOUTFILE, much prettier than ">/dev/null" Also introducing the -s (--silent) option to inhibit the ugly progress metric

curl -w "http/%{http_version} %{http_code} -- %{time_total}" -s -o/dev/null http://example.com

slide-43
SLIDE 43

FOR EXAMPLE: FOR EXAMPLE:

Introducing -oOUTFILE, much prettier than ">/dev/null" Also introducing the -s (--silent) option to inhibit the ugly progress metric

curl -w "http/%{http_version} %{http_code} -- %{time_total}" -s -o/dev/null http://example.com

We can also mention --stderr to control the error output

slide-44
SLIDE 44

FOR EXAMPLE: FOR EXAMPLE:

Introducing -oOUTFILE, much prettier than ">/dev/null" Also introducing the -s (--silent) option to inhibit the ugly progress metric

curl -w "http/%{http_version} %{http_code} -- %{time_total}" -s -o/dev/null http://example.com

We can also mention --stderr to control the error output Use with - to direct it to stdout

slide-45
SLIDE 45

FOR EXAMPLE: FOR EXAMPLE:

Introducing -oOUTFILE, much prettier than ">/dev/null" Also introducing the -s (--silent) option to inhibit the ugly progress metric

curl -w "http/%{http_version} %{http_code} -- %{time_total}" -s -o/dev/null http://example.com

We can also mention --stderr to control the error output Use with - to direct it to stdout Or whatever filename

slide-46
SLIDE 46

FOR EXAMPLE: FOR EXAMPLE:

Introducing -oOUTFILE, much prettier than ">/dev/null" Also introducing the -s (--silent) option to inhibit the ugly progress metric

curl -w "http/%{http_version} %{http_code} -- %{time_total}" -s -o/dev/null http://example.com

We can also mention --stderr to control the error output Use with - to direct it to stdout Or whatever filename >15 years using shells, still can't handle std flows ?

slide-47
SLIDE 47

FOR EXAMPLE: FOR EXAMPLE:

Introducing -oOUTFILE, much prettier than ">/dev/null" Also introducing the -s (--silent) option to inhibit the ugly progress metric

curl -w "http/%{http_version} %{http_code} -- %{time_total}" -s -o/dev/null http://example.com

We can also mention --stderr to control the error output Use with - to direct it to stdout Or whatever filename >15 years using shells, still can't handle std flows ? curl got your back.

slide-48
SLIDE 48

DID YOU EVER ? DID YOU EVER ?

slide-49
SLIDE 49

DID YOU EVER ? DID YOU EVER ?

…

slide-50
SLIDE 50

DID YOU EVER ? DID YOU EVER ?

… Need to edit /etc/hosts ?

slide-51
SLIDE 51

DID YOU EVER ? DID YOU EVER ?

… Need to edit /etc/hosts ?

curl -v --resolve www.example.com:443:1.2.3.4 https://www.example.com/

slide-52
SLIDE 52

DID YOU EVER ? DID YOU EVER ?

… Need to edit /etc/hosts ? No need to play around with "Host" header

curl -v --resolve www.example.com:443:1.2.3.4 https://www.example.com/

slide-53
SLIDE 53

MEMORY ALLOCATION MEMORY ALLOCATION PROBLEMS PROBLEMS

slide-54
SLIDE 54

MEMORY ALLOCATION MEMORY ALLOCATION PROBLEMS PROBLEMS

  • No. Don't.
slide-55
SLIDE 55

MEMORY ALLOCATION MEMORY ALLOCATION PROBLEMS PROBLEMS

  • No. Don't.

All the options I mentioned can be added to $HOME/.curlrc

slide-56
SLIDE 56

MEMORY ALLOCATION MEMORY ALLOCATION PROBLEMS PROBLEMS

  • No. Don't.

All the options I mentioned can be added to $HOME/.curlrc Or write several of these, and recall them with -K filename, or

  • -config
slide-57
SLIDE 57

CURL PLAYS NICE WITH CURL PLAYS NICE WITH OTHERS OTHERS

Or you can avoid the options madness and ordering, by just right- clicking in Firefox (and Chrome) DevTools.

slide-58
SLIDE 58

CURL PLAYS NICE WITH CURL PLAYS NICE WITH OTHERS OTHERS

Or you can avoid the options madness and ordering, by just right- clicking in Firefox (and Chrome) DevTools. And select "Copy as cURL"

slide-59
SLIDE 59

CURL PLAYS NICE WITH CURL PLAYS NICE WITH OTHERS OTHERS

Or you can avoid the options madness and ordering, by just right- clicking in Firefox (and Chrome) DevTools. And select "Copy as cURL" It works in BurpSuite too.

slide-60
SLIDE 60

CURL PUTS THE C IN CURL. CURL PUTS THE C IN CURL.

> curl https://example.com \

  • -header "Hello: World" -w '# %{http_code} -- %{time_total}'\
  • -libcurl test.c -so/dev/null

# 200 -- 0,339792% > cat test.c

slide-61
SLIDE 61

TEST.C TEST.C

/********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; struct curl_slist *slist1; slist1 = NULL; slist1 = curl_slist_append(slist1, "Hello: World"); hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, slist1);

slide-62
SLIDE 62

YOUR OWN STRESS-TEST YOUR OWN STRESS-TEST

Because after all, they're just glorified (yet customisable) loops with precise metrics Let's roll our own apache-bench

slide-63
SLIDE 63

#include <curl/curl.h> #include <omp.h> #define MAX_THREAD 64 #define LASERS 1000 #define URL "http://www.example.com" int main(int argc, char *argv[]) { int tid, i = 0; FILE *devnull; devnull = fopen("/dev/null", "w"); #pragma omp parallel private(i) num_threads(MAX_THREAD) { #pragma omp for for(i = 0; i < LASERS; ++i) { tid = omp_get_thread_num(); CURLcode ret; CURL *hnd; double total; curl_off_t dl, rate;

slide-64
SLIDE 64

Just removing some comments

#include <curl/curl.h> #include <omp.h> #define MAX_THREAD 64 #define LASERS 1000 #define URL "http://www.example.com" int main(int argc, char *argv[]) { int tid, i = 0; FILE *devnull; devnull = fopen("/dev/null", "w"); #pragma omp parallel private(i) num_threads(MAX_THREAD) { #pragma omp for for(i = 0; i < LASERS; ++i) { tid = omp_get_thread_num(); CURLcode ret; CURL *hnd; double total; curl_off_t dl, rate;

slide-65
SLIDE 65

Just removing some comments And wrap some OpenMP magic around

#include <curl/curl.h> #include <omp.h> #define MAX_THREAD 64 #define LASERS 1000 #define URL "http://www.example.com" int main(int argc, char *argv[]) { int tid, i = 0; FILE *devnull; devnull = fopen("/dev/null", "w"); #pragma omp parallel private(i) num_threads(MAX_THREAD) { #pragma omp for for(i = 0; i < LASERS; ++i) { tid = omp_get_thread_num(); CURLcode ret; CURL *hnd; double total; curl_off_t dl, rate;

slide-66
SLIDE 66

Just removing some comments And wrap some OpenMP magic around Compile with: gcc mt_curl.c -fopenmp -lcurl

#include <curl/curl.h> #include <omp.h> #define MAX_THREAD 64 #define LASERS 1000 #define URL "http://www.example.com" int main(int argc, char *argv[]) { int tid, i = 0; FILE *devnull; devnull = fopen("/dev/null", "w"); #pragma omp parallel private(i) num_threads(MAX_THREAD) { #pragma omp for for(i = 0; i < LASERS; ++i) { tid = omp_get_thread_num(); CURLcode ret; CURL *hnd; double total; curl_off_t dl, rate;

slide-67
SLIDE 67

Just removing some comments And wrap some OpenMP magic around Compile with: gcc mt_curl.c -fopenmp -lcurl Make sure the entire file is <42 LoC

#include <curl/curl.h> #include <omp.h> #define MAX_THREAD 64 #define LASERS 1000 #define URL "http://www.example.com" int main(int argc, char *argv[]) { int tid, i = 0; FILE *devnull; devnull = fopen("/dev/null", "w"); #pragma omp parallel private(i) num_threads(MAX_THREAD) { #pragma omp for for(i = 0; i < LASERS; ++i) { tid = omp_get_thread_num(); CURLcode ret; CURL *hnd; double total; curl_off_t dl, rate;

slide-68
SLIDE 68

DEMO ? DEMO ?

slide-69
SLIDE 69

THANK YOU THANK YOU

Everyone @ PTS for all the event Dan Stanberg for all of the curling Have a safe trip back home ♥

slide-70
SLIDE 70
slide-71
SLIDE 71

QUESTIONS ? QUESTIONS ?