Problem with texture masking

I am trying to do a spotlight effect based on the users mouse position, my problem is I am getting jagged edges. Please help me. The mask is made from drawing to an FBO

When I Texture the plane with the mask it looks fine:

but then when I use this mask, notice the circle becomes jagged:

Here is the frag shader I am using:

#version 330

in vec2 colorCoord;

uniform sampler2D colorTexture;
uniform sampler2D maskTexture;

out vec4 outputColor;

void main(){
	
	vec4 maskColor = texture(maskTexture,colorCoord);
	vec4 black = vec4(0,0,0,1);
	vec4 yellow = vec4(1,1,0,1);
	vec4 textureColor = texture(colorTexture,colorCoord);
	if(maskColor.r > 0.0 || maskColor.g > 0.0 || maskColor.b > 0.0){
		textureColor.w = 1.0;
		outputColor = mix(textureColor,yellow,.5);
	}else{
		textureColor.w = .2;
		outputColor = mix(textureColor,black,.8);
	}
	
}

The mask is made from drawing to an FBO.I created the fbo with 8 samples.

    mask.allocate(500, 500,GL_RGBA,8);

ok I found the problem. In the mask texture not all of the pixels are completely white. Changing the fragment shader to this fixed the problem.

#version 330

in vec2 colorCoord;

uniform sampler2D colorTexture;
uniform sampler2D maskTexture;

out vec4 outputColor;

void main(){
	
	vec4 maskColor = texture(maskTexture,colorCoord);
	vec4 black = vec4(0,0,0,1);
	vec4 yellow = vec4(1,1,0,1);
	vec4 textureColor = texture(colorTexture,colorCoord);
	
	vec4 insideColor = mix(textureColor,yellow,.5); 
	insideColor.w = 1.0;
	vec4 outsideColor = mix(textureColor,black,.8);
	outsideColor.w = 1.0;
	if(maskColor.r > 0.0){
		outputColor = mix(outsideColor,insideColor,maskColor.r);
	}else{
		outputColor = outsideColor;
	}
	
}

Since you are already using a shader and fbo, you could generate your circle directly in GLSL too. Pass a vec2 uniform representing the center of the circle, and uniform float representing the radius. The use a distance function to see if you are inside the circle (white) or not (black). You could then easily modify this spotlight to have soft edges by using two radii, and interpolating between them for a gradient effect.