Deleting a specific user voice from the original audio track with Python

Diarization, also known as speaker diarization, is the process of automatically identifying and separating different speakers in an audio recording. This involves analyzing the acoustic characteristics of the voices, such as pitch, tone, and rhythm, and clustering them into distinct speaker groups. Diarization is used in a variety of applications, including transcription, meeting analysis, and surveillance. 
 
pyannote.audio is a Python library for audio analysis, including speaker diarization. It offers pre-trained models for speaker embedding, clustering, and segmentation, allowing you to automatically separate audio recordings into segments corresponding to different speakers. You can adjust parameters like the number of speakers and the model type to customize the diarization process. Additionally, you can access pre-trained models for other tasks like speaker verification and voice activity detection. 

Here's a step-by-step guide using pyannote.audio to perform speaker diarization:

Step 1: Install pyannote.audio and Dependencies

You'll need Python 3.8 or 3.9 for compatibility with pyannote.audio. Install the library along with the required dependencies.

pip install pyannote.audio
pip install torch torchvision torchaudio  # for PyTorch (use the right version for your OS)
pip install huggingface_hub

Step 2: Authenticate with Hugging Face

pyannote.audio uses pre-trained models hosted on Hugging Face, so you'll need an access token. You can create one by signing up at Hugging Face and generating a token.

from huggingface_hub import login
login(token="YOUR_HUGGING_FACE_ACCESS_TOKEN")

Step 3: Perform Speaker Diarization

Here's a simple example that loads an audio file, performs diarization, and outputs the timestamps of each speaker's segments.

from pyannote.audio import Pipeline

# Initialize the speaker diarization pipeline
pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization")

# Apply the pipeline to an audio file
diarization_result = pipeline("your_audio_file.wav")

# Display the speaker segments
for turn, _, speaker in diarization_result.itertracks(yield_label=True):
    print(f"Speaker {speaker}: from {turn.start:.1f}s to {turn.end:.1f}s")

The output will provide time intervals during which each speaker is talking. For example:

Speaker 1: from 0.0s to 5.5s
Speaker 2: from 5.5s to 10.3s
Speaker 1: from 10.3s to 15.0s

Step 4: Remove or Modify Specific Speaker Segments

Once you have the timestamps, you can use pydub to load the audio, split it by speaker segments, and then delete or replace specific speaker sections.

Here's an example of removing all segments spoken by "Speaker 1":

from pydub import AudioSegment

# Load the original audio
original_audio = AudioSegment.from_file("your_audio_file.wav")

# Initialize an empty audio segment
output_audio = AudioSegment.empty()

# Add only non-Speaker 1 segments to the output audio
for turn, _, speaker in diarization_result.itertracks(yield_label=True):
    if speaker != "SPEAKER_01":
        segment = original_audio[turn.start * 1000 : turn.end * 1000]
        output_audio += segment

# Export the modified audio
output_audio.export("modified_audio_no_speaker_1.wav", format="wav")

This example removes all segments where Speaker 1 is detected. You can replace segments or add effects similarly, using the timestamps to manipulate the audio selectively.

Below case keeps the original duration unlike the above example:

from pydub import AudioSegment

# Load the original audio
original_audio = AudioSegment.from_file("your_audio_file.wav")

# Initialize an empty audio segment with silence
output_audio = AudioSegment.silent(duration=len(original_audio))

# Add only non-Speaker 1 segments to the output audio
for turn, _, speaker in diarization_result.itertracks(yield_label=True):
    if speaker != "SPEAKER_01":
        segment = original_audio[turn.start * 1000 : turn.end * 1000]
        output_audio = output_audio.overlay(segment, position=turn.start * 1000)

# Export the modified audio
output_audio.export("modified_audio_no_speaker_1.wav", format="wav")

Explanation

• AudioSegment.silent(duration=len(original_audio)) creates an output with the same length as the original but filled with silence.
• output_audio.overlay(segment, position=turn.start * 1000) adds the non-Speaker 1 segments at the correct positions while preserving silence in the removed sections.

This should retain the original duration of the audio file.