Retrieves archived tweets CDX data from the Wayback Machine, performs necessary parsing (see Field Options), and saves the data in HTML, for easy viewing of the tweets using the iframe tags, CSV, and JSON formats.
It is compatible with Python versions 3.10 and above. See installation options.
pipx install waybacktweets
Usage: waybacktweets [OPTIONS] USERNAME USERNAME: The Twitter username without @ Options: -c, --collapse [urlkey|digest|timestamp:xx] Collapse results based on a field, or a substring of a field. XX in the timestamp value ranges from 1 to 14, comparing the first XX digits of the timestamp field. It is recommended to use from 4 onwards, to compare at least by years. -f, --from DATE Filtering by date range from this date. Format: YYYYmmdd -t, --to DATE Filtering by date range up to this date. Format: YYYYmmdd -l, --limit INTEGER Query result limits. -rk, --resumption_key TEXT Allows for a simple way to scroll through the results. Key to continue the query from the end of the previous query. -mt, --matchtype [exact|prefix|host|domain] Results matching a certain prefix, a certain host or all subdomains. -v, --verbose Shows the log. --version Show the version and exit. -h, --help Show this message and exit. Examples: waybacktweets jack waybacktweets --from 20200305 --to 20231231 --limit 300 --verbose jack Repository: https://github.com/claromes/waybacktweets Documentation: https://waybacktweets.claromes.com
from waybacktweets import WaybackTweets, TweetsParser, TweetsExporter USERNAME = "jack" api = WaybackTweets(USERNAME) archived_tweets = api.get() if archived_tweets: field_options = [ "archived_urlkey", "archived_timestamp", "parsed_archived_timestamp", "archived_tweet_url", "parsed_archived_tweet_url", "original_tweet_url", "parsed_tweet_url", "available_tweet_text", "available_tweet_is_RT", "available_tweet_info", "archived_mimetype", "archived_statuscode", "archived_digest", "archived_length", "resumption_key", ] parser = TweetsParser(archived_tweets, USERNAME, field_options) parsed_tweets = parser.parse() exporter = TweetsExporter(parsed_tweets, USERNAME, field_options) exporter.save_to_csv() exporter.save_to_json() exporter.save_to_html()
A prototype written in Python with the Streamlit framework and hosted on Streamlit Cloud.
Important: Starting from version 1.0, the web app will no longer receive all updates from the official package. To access all features, prefer using the package from PyPI.
- Tristan Lee (Bellingcat's Data Scientist) for the idea.
- Jessica Smith (Snowflake's Community Growth Specialist) and Streamlit team for the additional server resources on Streamlit Cloud.
- OSINT Community for recommending the package and the application.