A real-time news sentiment analysis ETL pipeline built with Pathway, designed to fetch, process, and analyze news articles for sentiment insights.
- Real-time Processing: Stream-based processing using Pathway
- Multiple Data Sources: Support for NewsAPI and RSS feeds
- Advanced Sentiment Analysis: Using both TextBlob and Transformers models
- Flexible Output: Store results in various formats (JSON, CSV, database)
- Scalable Architecture: Built for high-throughput processing
-
Install Dependencies:
pip install -r requirements.txt
-
Set up Environment Variables:
cp .env.example .env # Edit .env with your API keys -
Run the Pipeline:
python main.py
├── main.py # Main pipeline entry point
├── src/
│ ├── data_sources/ # News data source connectors
│ ├── processors/ # Data processing and transformation
│ ├── sentiment/ # Sentiment analysis modules
│ └── outputs/ # Output handlers
├── config/
│ └── settings.py # Configuration settings
├── tests/ # Unit tests
└── data/ # Output data storage
The pipeline can be configured through environment variables or the config/settings.py file:
NEWS_API_KEY: Your NewsAPI keySENTIMENT_MODEL: Choice of sentiment model (textblob/transformers)OUTPUT_FORMAT: Output format (json/csv/database)BATCH_SIZE: Processing batch sizeUPDATE_INTERVAL: How often to fetch new data
from main import NewsSentimentPipeline
pipeline = NewsSentimentPipeline()
pipeline.run()from main import NewsSentimentPipeline
from config.settings import Settings
settings = Settings(
news_api_key="your_key",
sentiment_model="transformers",
output_format="json"
)
pipeline = NewsSentimentPipeline(settings)
pipeline.run()The pipeline outputs sentiment analysis results in the following format:
{
"article_id": "unique_id",
"title": "Article Title",
"content": "Article content...",
"source": "news_source",
"published_at": "2024-01-01T00:00:00Z",
"sentiment_score": 0.75,
"sentiment_label": "positive",
"confidence": 0.92,
"processed_at": "2024-01-01T00:00:00Z"
}- Fork the repository
- Create a feature branch
- Make your changes
- Add tests
- Submit a pull request
MIT License