Today, we’re launching new batch compliance endpoints to make it easier for developers using the Twitter API to keep their Twitter data in compliance with our Developer Agreement and Policy. These new endpoints allow developers and researchers to batch upload large amounts of Twitter data and understand what action is needed to ensure that their datasets reflect user intent and the current state of the content on Twitter.
The endpoints will be particularly helpful to those who work with large amounts of Twitter data, whether to conduct academic research, social listening, social analytics, or any other approved developer use case. Notably, today’s launch does not include changes to the Developer Agreement and Policy, nor does it change Twitter’s expectations of developers storing data. Rather, this launch provides a new, additional solution to help all developers keep Twitter data up to date with the current state of content on our service. Please note: the use of the batch compliance endpoints is restricted to the aforementioned use cases, and any other purpose is prohibited and may result in enforcement action.
How does it work?
The batch compliance endpoints allow developers to submit a set of Tweet IDs or user IDs to check the current status of those Tweets and users on Twitter and to determine what actions are required to update their datasets. After uploading the Tweet and user IDs, these endpoints will return the corresponding IDs that require developer action, as well as reason for that action and metadata to support the required action. For example, when a Tweet has been deleted on Twitter, the developer will be instructed to delete this Tweet from their dataset as well.
Typically, there are 4 steps involved in working with these endpoints:
1. Create a compliance job
You can specify the job type (with the value ‘tweets’ or ‘users’ to indicate whether the dataset you want to upload has Tweet IDs or user IDs. Note: You can have one concurrent job per job type at any time.
2. Upload your dataset to the upload_url
Next, you upload your dataset as a plain text file to the provided upload_url, with each line of the file containing a single Tweet ID or user ID. Note: The upload_url expires after 15 minutes.
3. (Optional) Check the job status
You can check the status of your compliance job to see whether it is created, in_progress, failed, expired or complete.
4. Download the results
When your job is complete, you can download the results using the download_url. Note: The download_url expires after one week (from when the job was created).
This result will contain a set of JSON objects (one object per line). Each object will contain a Tweet ID, the Tweet’s creation date (useful to locate Tweets organized by date), required action, the reason for the compliance action, and its date, as shown below:
{"id":"1265324480517361664","action":"delete","created_at":"2019-10-29T17:02:47.000Z","redacted_at":"2020-07-29T17:02:47.000Z","reason":"deleted"}
{"id":"1263926741774581761","action":"delete","created_at":"2019-10-29T17:02:47.000Z","redacted_at":"2020-07-29T17:02:47.000Z","reason":"protected"}
{"id":"1265324480517361669","action":"delete","created_at":"2019-10-29T17:02:47.000Z","redacted_at":"2020-07-29T17:02:47.000Z","reason":"suspended"}
The compliance events (indicated by the reason in the response above) will be from the following:
deleted (indicating that the Tweet or User account was deleted)deactivated (indicating that the Tweet or User account has been deactivated)scrub_geo (indicating that the geo information associated with the Tweet or User was removed)protected (indicating the account that made the Tweet became private)suspended (indicating the account that made the Tweet was suspended)
Getting started
In order to get started with these endpoints, you can use the quick start guide. We also have code samples available in Python, JavaScript etc. in our TwitterDev Github repository to help you get started quickly. Just add the appropriate Bearer Token from your App to the code samples in order to connect to this endpoint. If you use Python, check out this utility that makes it easy for you to work with these endpoints.
What’s next?
We’re exploring adding additional support for compliance events like:
- Tweet drops
- Tweet and user withhelds
We hope that this endpoint will make it easy for developers and researchers to keep their dataset compliant. If you have questions about this endpoint, feel free to ask in our Twitter community forums