How to use Twitter’s search API with 4D

Given that there are around 6,000 tweets every second (i.e. over 350,000 tweets per minute, 500 million tweets per day equaling approximately 200 billion tweets per year), it would be great if we could search for those related to a specific subject rather than an avalanche of tweets we’re not interested in. Good news! Twitter provides a REST Search API that is great for searching tweets.

In this post we’ll discuss how to use Twitter’s Search API with 4D (Jump to the database example).

twitter access tokens

To use Twitter’s Search API, you need access tokens. This is mandatory to send requests to the API. In order to get these tokens, you must create a Twitter Application (a free, but necessary step):

1. Create a Twitter account (if you don’t already have one).
2. Go to https://dev.twitter.com/apps/new and log in.
3. Click the “Create New App” button, fill out the form, agree to the terms and click the “Create your Twitter application” button.
4. Next, on the “Keys and Access Tokens” tab, copy your “API key” and “API secret”. 

   

5. Scroll down and click on “Create my access token”. Copy your “Access token” and “Access token secret”. By doing so, you give your Twitter account access to use this API. Then you’ll see:

   

Note: These tokens should be considered as sensitive as passwords and must not be shared or distributed to untrusted parties.

authentication

Now that we have our tokens, let’s get our hands dirty and write the authentication code. For this, we’ll use the “Application-only authentication” process. This lets an application issue authenticated requests for itself, and not on behalf of a specific user. This process requires an application to:

• encode its consumer key and secret.
• make a request to the POST oauth2 / token endpoint to exchange these credentials for a bearer token.
• use this bearer token for authentication when accessing the REST API.

Here’s how you do it with 4D:

• Concatenate the consumer key and the encoded consumer secret into a single string with a colon character ”:”.
• Convert your string into a BLOB with the TEXT TO BLOB command.
• Encode the blob with the BASE64 ENCODE command.

SET BLOB SIZE(vxBlob;0)
$consumerKey:="your consumer key here" 
$consumerSecret:="your consumer secret here"
TEXT TO BLOB($consumerKey+":"+$consumerSecret; vxBlob; UTF8 text without length)
BASE64 ENCODE(vxBlob; $authBasicTxt)

• The resulting value must be exchanged for a bearer token by issuing a request to POST oauth2 / token. This request is made with the 4D HTTP Request command and the HTTP POST method selector. It must include an Authorization header and a Content-Type, with the body of the request as grant_type=client_credentials.

$authRequestContent:="grant_type=client_credentials"
ARRAY TEXT($arrHeadAttr;0)
ARRAY TEXT($arrHeadValue;0)
APPEND TO ARRAY($arrHeadAttr;"Authorization")
APPEND TO ARRAY($arrHeadAttr;"Content-Type")
APPEND TO ARRAY($arrHeadValue;"Basic "+$authBasicTxt)
APPEND TO ARRAY($arrHeadValue;"application/x-www-form-urlencoded;charset=UTF-8")
$url:="https://api.twitter.com/oauth2/token" 
C_TEXT($response) 
$result:=HTTP Request(HTTP POST method;$url;$authRequestContent;$response; $arrHeadAttr;$arrHeadValue;*)

• Make sure your HTTP request is successful, by verifying that the response status code is 200. (More on HTTP status codes)
• Parse the HTTP response with the JSON Parse command, then store the value of the access_token property into a variable.

C_OBJECT($jsonResponse) 
$jsonResponse:=JSON Parse($response)
$accessToken:=OB Get($jsonResponse;"access_token";Is text)


• This bearer token will be used to issue a request to our search API (details in search for tweets section). So copy it and keep it close.

search for tweets

The complicated part is over. Now that we have our access token, we can use it to issue a request to GET search/tweets.json:

$accessToken:="Your access token from the authentication section"
ARRAY TEXT($searchArrHeadAttr;0)
ARRAY TEXT($searchArrHeadValue;0)
APPEND TO ARRAY($searchArrHeadAttr;"Authorization")
APPEND TO ARRAY($searchArrHeadValue;"Bearer "+$accessToken)

$query:="peace"
$url:="https://api.twitter.com/1.1/search/tweets.json?q="+$query
$result:=HTTP Request(HTTP GET method;$url;$searchBody;$searchReponse;$searchArrHeadAttr;$searchArrHeadValue;*)


In the debugger, drag your $searchResponse variable into the Expression window, and you’ll notice that the result contains keywords matching our query. 

Below is a simplified JSON response for the “peace” keyword:

{  
   "statuses":[  
      {  
         "created_at":"Tue Nov 21 09:51:09 +0000 2017",
         "id":932909235528183808,
         "id_str":"932909235528183808",
         "text":"RT @AraiEij: @pintsize73 \\nMay your birthday and every day be 
          filled with love, peace and joy",
         ....
         "retweeted_status":{  
            "created_at":"Sun Nov 19 05:48:49 +0000 2017",
            "id":932123476025946112,
            "id_str":"932123476025946112",
            "text":"@pintsize73 Happy Birthday dear Marie \\nMay 
             your birthday and every day be filled with love, peace",
            "truncated":true,
            "entities":{  
               "hashtags":[],
               "user_mentions":[  
                  {...}
               ],
               "urls":[]
            },
            "metadata":{  
               "iso_language_code":"en",
               "result_type":"recent"
            }
         }
      }
   ],
   "search_metadata":{  
      ...
      "query":"Peace",
      "refresh_url":"?since_id=932909235528183808&q=Peace&include_entities=1",
      ...}
}

And here is a sneak peek of the search result UI: 

See more with our example

Check out our application example! Please note that this example requires that you follow some of the steps described. In fact, you will need to get your twitter access tokens to run the demo.

Example: How to use Twitter’s Search API