# CIMRI.COM Price Scraper Implementation

## Overview

This implementation scrapes price data from cimri.com for StockItem records where `min_quantity` is not null, and stores the first 4 search results in the PriceHistory table.

## Components Created

### 1. CimriComScraper Class

**File:** `app/Services/Scrapers/CimriComScraper.php`

Main scraper that handles:

-   Searching products by name on cimri.com
-   Parsing search results to extract product information
-   Extracting first 4 product results with prices

### 2. CimriPriceService Class

**File:** `app/Services/CimriPriceService.php`

Orchestration service that handles:

-   Filtering StockItems with `min_quantity > 0`
-   Managing the scraping process
-   Saving results to PriceHistory table
-   Duplicate prevention
-   Rate limiting (2-second delays)

### 3. Artisan Command

**File:** `app/Console/Commands/ScrapeCimriPrices.php`

Command-line interface for running the scraper:

```bash
# Process all StockItems with min_quantity > 0
php artisan scrape:cimri-prices

# Test with specific product
php artisan scrape:cimri-prices --test="ayçiçek yağı"

# Process with limit
php artisan scrape:cimri-prices --limit=10

# Skip items without names
php artisan scrape:cimri-prices --skip-without-name
```

### 4. Database Migration

**File:** `database/migrations/2026_02_18_000000_insert_cimri_com_price_source.php`

Creates the PriceSource record for CIMRI.com in the database.

### 5. Test Script

**File:** `test_cimri_scraper.php`

Standalone test script to verify functionality without Artisan commands.

## How It Works

1. **Data Selection**: Queries StockItem records where `min_quantity IS NOT NULL AND min_quantity > 0`
2. **Search Process**: For each item, searches cimri.com using the item name
3. **URL Construction**: `https://www.cimri.com/market/arama?q={product_name}&sort=price-asc`
4. **Data Extraction**: Parses the search results to get first 4 products with prices
5. **Storage**: Saves each result to PriceHistory table with proper associations

## Key Features

-   **Rate Limiting**: 2-second delays between requests to avoid blocking
-   **Duplicate Prevention**: Checks for existing entries within 24 hours
-   **Error Handling**: Comprehensive logging and error recovery
-   **Flexible Options**: Command-line flags for testing and limiting
-   **Database Integration**: Proper foreign key relationships maintained

## Usage Examples

### Basic Usage

```bash
php artisan scrape:cimri-prices
```

### Testing Specific Products

```bash
php artisan scrape:cimri-prices --test="ayçiçek yağı"
php artisan scrape:cimri-prices --test="süt"
```

### Limited Processing

```bash
# Process only first 5 items
php artisan scrape:cimri-prices --limit=5
```

### Production Run

```bash
# Full production run with all safety measures
php artisan scrape:cimri-prices --skip-without-name
```

## Database Structure

The implementation uses these existing tables:

-   **StockItem** (mysql-remote): Source data with product names
-   **PriceHistory** (mysql): Destination for scraped price data
-   **PriceSource** (mysql): Metadata about the scraping source

## Error Handling

-   Logs all scraping attempts and results
-   Continues processing even if individual items fail
-   Provides detailed error reporting
-   Handles network timeouts and parsing errors gracefully

## Performance Considerations

-   Built-in delays prevent rate limiting
-   Batch processing with configurable limits
-   Efficient database queries
-   Memory-conscious processing for large datasets

## Testing

Run the standalone test script:

```bash
php test_cimri_scraper.php
```

This will:

1. Test the scraper with sample products
2. Verify database integration
3. Show sample results without permanent storage

## Dependencies

-   PHP DOM extension (for HTML parsing)
-   Laravel HTTP Client (included in Laravel)
-   Existing database connections (mysql and mysql-remote)

## Notes

-   The scraper respects cimri.com's structure as seen in CIMRI-COM.html
-   Implements proper user-agent and headers for compatibility
-   Handles Turkish character encoding properly
-   Designed to be extensible for other price sources