I always say “A Template is always someone else’s template”. It is almost always impossible to use a template as-is. However good a template, tool or hack might be, but it has to be tweaked to suit your need. I have been using several tools throughout my academic and professional journey. The list has evolved and many of the tools and templates have been replaced by better alternatives as the world yielded to my needs :).
Tools for Logging, Tracking & Research
I have used many Agile tools and ways to organise my thoughts and plans. I am fanatic about documenting and organising my ideas. As a result I have ended up with scores of tracks and hundreds of items in each of them. I grew from physical notebooks, organisers, dashboards to dynamic Agile boards.
- swiftkey — Ever since I started swiping, I can’t type anymore on my phone
- google keep — satisfies my simple need – notes and accessibility.
- evernote — best for longer notes and is also cross platform. I have reduced my use since I now prefer jumping directly from notes to dashboard
- asana — A easy to use task management tool and much more. It has solved many of my needs. A great launching pad for my notes to my projects.
- ES file explorer – The file explorer which does it all
- Onedrive and google drive – for accessing my imp stuff across devices. The free storage is enough for my needs.
- google photos – Enough for all my photos across my devices.
- google colab – a recent but wonderful addition which has made by data science coding easy and platform/device independent.
Never type but swype
I am required to participate in the cause (meetings, chats , seminars), take minutes, but also want to jot down any ideas that I might want to pursue later. This dynamic environment required speed. I used physical notes till 2015, because nothing beats the speed of a quick scribble.
Things changed when Swype came in for android smart phones and allowed me to swipe and not type my notes. This was fast and by the year end I was able to swipe faster than I could type. I can now swipe faster than taking shorthand notes. This also saved me the time needed to convert my paper notes to digital.
Sadly swype died in Feb 2018 and I am having to survive with swiftkey. I miss swype keyboard but have to live with what is available. I can’t and don’t want to type anymore on a phone. The next transition – I am still waiting when speech dictation is satisfactory.
Sep-2020 Update – With the COVID-19 lockdown in place I do not need to rely on my phone as much. I mostly use my microsoft surface. Hence keyboard typing has taken over swiping.
Notes , Idea Collectors (or hoarder!!)
I use google keep, asana and wordpress P2 to aid my journey of notes, task arrangement and implementation.
Tools for Data Science & Research
pycharm/colab
My programming language of choice are Excel Macro and Python. Nowadays I only code in python and I am a big fan of the google colab environment. Colab is similar to a local Jupyter environment but takes away the trouble of system configuration. Since I can save my colab notebooks in google drive or github, I am also device independent. It is easy for me to move across my devices. It is actually cool.
I intent to post eventually on GitHub or here in my blog. Colab notebooks make it easy to build up an article directly from notebooks.
The rest of the page lists the tools and links that I have collected during my ongoing study. I have not used all of them yet but intend to, during this year. I will be sorting and providing my comments as I deal with them.
Data collection
- twg — Python script, leveraging Tweepy, for grabbing tweets from the Twitter streaming API
- Back to Future (btf) tweet collector — tweet searching and streaming.
- GetOldTweets3 — Scrape (historical) twitter data without using the API. Make sure to follow the Twitter ToS.
- fb_scrape_public — downloads public data from Facebook (Python package by Deen Freelon).
- Facebook Group Data Downloader — batch downloads with fb_scrape_public.
- instaloader — powerful and easy-to-use Python API for Instagram (Python package by André Koch-Kramer).
YouTube
- youtube-comment-scraper – downloads the comments to YouTube videos (Node.js, by Philip Klostermann).
Web
- wget — powerful tool, of hacking fame, to use with caution.
Data wrangling
File management
- txtls — Python scripts to do stuff with text data.
Data cleaning
- TXT PURIFIER — notebook for cleaning documents before analysis (Python).
Parsing and tagging
- LXML HTML parsing — notebook about how to parse locally stored html files (Python).
- py-json-parser — notebook on parsing JSON files into pandas (Python).
- JSON-parser — notebook on parsing JSON files into dataframes (R).
- efselab — powerful text tagger (compiler by Robert Östling).
Data analysis
Text analysis
- Textometrica — tool for Connected Concept Analysis (online service/PHP).
- ThingLinker — script for Actor-Network analysis of texts (Python).
- Topic modelling with MALLET — an introduction to how to fit LDA topic models to documents (Java).
- Topic Modelling Tool — A GUI for MALLET’s implementation of LDA (app by Scott Enderle).
- Topic top documents — extracting the most likely documents for MALLET topics (Python).
- Document distances and clustering — calculates and visualises similarity/distance between texts (Python).
- Sentiment analysis with VADER — produces scores for emotional content in social media texts (Python).
Social network analysis