# Anti-Ban Measures for Market Price Scraping

This document explains the anti-ban measures implemented to prevent IP blocking when scraping market prices from marketkarsilastir.com.

## Implemented Measures

### 1. Rate Limiting

-   **Individual Request Delays**: Random delays between 2-5 seconds between each request
-   **Batch Processing**: Items processed in batches of 10 with 30-second delays between batches
-   **Purpose**: Mimics human browsing behavior to avoid detection

### 2. User Agent Rotation

-   Multiple realistic browser user agents are rotated randomly
-   Includes Chrome, Firefox, and Safari user agents from different platforms
-   Makes requests appear to come from different browsers/devices

### 3. Realistic HTTP Headers

-   Proper Accept headers mimicking real browser requests
-   Language preferences set to Turkish/English
-   Connection keep-alive and upgrade headers

### 4. Retry Mechanism

-   Automatic retry up to 3 times on failure
-   10-second delay between retry attempts
-   Prevents temporary network issues from causing complete failure

### 5. Proxy Support (Configurable)

-   Optional proxy configuration for IP rotation
-   Can be enabled in `config/scraping.php` when needed
-   Allows distributing requests across multiple IP addresses

### 6. Off-Peak Scheduling

-   Scheduled to run at 2:00 AM daily
-   Reduces load on target server during peak hours
-   Uses `withoutOverlapping()` to prevent concurrent executions

## Configuration

All settings can be adjusted in `config/scraping.php`:

```php
'rate_limiting' => [
    'min_delay' => 2,      // Minimum delay between requests (seconds)
    'max_delay' => 5,      // Maximum delay between requests (seconds)
    'batch_size' => 10,    // Number of items per batch
    'batch_delay' => 30,   // Delay between batches (seconds)
],

'user_agents' => [
    // List of user agents to rotate through
],

'retry' => [
    'max_attempts' => 3,        // Maximum retry attempts
    'delay_between_retries' => 10, // Delay between retries (seconds)
],
```

## Usage

### Manual Execution

```bash
php artisan scrape:market-prices --limit=20
```

### Scheduled Execution

The command runs automatically daily at 2:00 AM as configured in `app/Console/Kernel.php`.

## Monitoring

The scraper logs detailed information including:

-   Processing statistics (success/failure counts)
-   Individual item processing results
-   Warning and error messages
-   Batch processing information

Check Laravel logs for monitoring: `storage/logs/laravel.log`

## Best Practices

1. **Start Conservative**: Begin with small batch sizes and longer delays
2. **Monitor Logs**: Regularly check logs for errors or warnings
3. **Adjust as Needed**: Increase delays if you notice issues
4. **Respect robots.txt**: Ensure compliance with target site policies
5. **Consider Caching**: Cache results to reduce frequency of requests

## Troubleshooting

If you encounter IP bans:

1. Increase delay times in configuration
2. Enable proxy support
3. Reduce batch sizes
4. Change execution schedule to less frequent intervals
5. Check if the target site has changed its structure

## Legal Notice

Always ensure your scraping activities comply with:

-   The website's Terms of Service
-   robots.txt guidelines
-   Local and international laws
-   Ethical web scraping practices