Indexing Facebook with Splunk - Part 1
This article shows an approach to acquire and index data from Facebook. Why? Facebook is a huge data set - and Splunk a tool to handle big data sets and enrich machine data with "Business Intelligence". In detail, I show how to index your list of friends on Facebook on a regular basis without writing one line of code.
What you gonna need:
- a Facebook account
- at least one friend on Facebook (otherwise, what's the point 
)
- a running splunk installation with decent permissions
- 10 minutes
Let's go:
Hit https://developers.facebook.com/apps and create a new app. Name it however you want. I guess you need to disable the sandbox mode. Note down the App ID and the App Secret.
Creating a new Facebook App which grants me access to the Graph API
Browse to Facebook Graph API Explorer. Switch the Application to the one you just created and click Get Acccess Token. There are no further permissions required so just click on Get Acccess Token. Hit Ok in the openend popup. So now you have a token which is valid for one hour. Click on Debug to get to the Access Token Debugger.
The default access token is only valid for an hour.
To make this token last longer, copy-paste the following URL in an editor and replace the three upper case place holders with the values you. As existing access token, use the one that we just created:
https://graph.facebook.com/oauth/access_token?client_id=APP_ID
&client_secret=APP_SECRET
&grant_type=fb_exchange_token
&fb_exchange_token=EXISTING_ACCESS_TOKEN
Open this URL in your browser and copy paste the resulting access token somewhere you don't loose it. To check that everything went fine, paste it in the token text box and click on Debug in the Graph API Explorer.
The exchanged token is now valid for 2 months.
In splunk, click on App -> Find more Apps -> Seach for "REST API Modular Input" -> Install free. When done installing, click on Manager -> Data Inputs -> REST -> New.
Adding a new Data Input to slpunk
Enter the following values:
REST API Input Name: MyFriendIndexer (whatever you like)
Endpoint URL: https://graph.facebook.com/me/friends
URL Arguments: access_token={SECOND TOKEN YOU GOT}
Response Type: json
Polling Interval: 3600 (1h)
Set sourcetype: Manual
Source type: json_no_timestamp
Hit Save. The first indexing is carried out instantly. Check whether everything is working by querying the following search:
source="rest://MyFriendIndexer"
Depending on the number of friends you have, you will see multiple logs. This is because Facebook does not deliver all the friends in one request, but appends a "next" URL to the end of the data. Luckily, this is directly handled by the REST API Modular Input source. To combine these logs, just modify the query slightly:
source="rest://MyFriendIndexer" | transaction maxpause=1s
Recover sudo abilities in debian
In case you screw up your sudo functionality, e.g. by removing yourself (the entity using sudo) from the sudoers using sudo, you can fix that easily using this step-by-step guide.
Splunk: Keep indexed volume under control
You can use splunk for free for up to 500 megabytes of indexed data daily. To make sure you do not run beyond this limitation, use the following query:
index=_internal group="per_source_thruput" NOT series="*splunk/var/log*" | eval mb=kb/1024| timechart span=1d sum(mb) by series
When drawing a bar graph, you directly see, what type of log is filling your indexes. In this case the "ps" logs.
Mount Remote Folders Via sshfs in OSX
http://www.danbishop.org/2011/09/10/how-to-mount-sftp-ssh-shares-in-os-x-lion/
Finding large files in Linux
Spring clean up? Find all the files which are larger than 20 megs:
find / -type f -size +20000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
Eclipse Juno and Retina Display on Mac OSX
Using Eclipse (Juno) on a retina Mac will disappoint you. Out of the box, all the fonts and symbols look like they're directly from the 90ies:
However, there is an easy way to fix that. Open a Finder window and chose Applications. There, find the installed eclipse application and right click it and chose Show Package Contents:
In the package, open Contents/Info.plist with an editor of your choice. In there, append the following 2 lines just before </dict>:
<key>NSHighResolutionCapable</key>
<true/>
Then, before restarting eclipse, drag-and-drop Eclipse into another folder and back to where it was. In my case, I moved into eclipse/readme and back to eclipse.
Et voila, Eclipse looks great now:
