1x1 Convolution. How does the math work?
Clash Royale CLAN TAG#URR8PPP
up vote
4
down vote
favorite
So I stumbled upon Andrew Ng course on 1x1 convolutions.
There he explains that you can use 1x1x192 convolution to shrink it.
But when I do
input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])
out = torch.mul(input_,filter)
I obviously get 28x28x192
matrix. So how should I be able to shrink it?
Just add the result of every 1x1x192 * 1x1x192
kerner result? So I'd get 28x28x1
matrix?
convnet
add a comment |Â
up vote
4
down vote
favorite
So I stumbled upon Andrew Ng course on 1x1 convolutions.
There he explains that you can use 1x1x192 convolution to shrink it.
But when I do
input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])
out = torch.mul(input_,filter)
I obviously get 28x28x192
matrix. So how should I be able to shrink it?
Just add the result of every 1x1x192 * 1x1x192
kerner result? So I'd get 28x28x1
matrix?
convnet
add a comment |Â
up vote
4
down vote
favorite
up vote
4
down vote
favorite
So I stumbled upon Andrew Ng course on 1x1 convolutions.
There he explains that you can use 1x1x192 convolution to shrink it.
But when I do
input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])
out = torch.mul(input_,filter)
I obviously get 28x28x192
matrix. So how should I be able to shrink it?
Just add the result of every 1x1x192 * 1x1x192
kerner result? So I'd get 28x28x1
matrix?
convnet
So I stumbled upon Andrew Ng course on 1x1 convolutions.
There he explains that you can use 1x1x192 convolution to shrink it.
But when I do
input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])
out = torch.mul(input_,filter)
I obviously get 28x28x192
matrix. So how should I be able to shrink it?
Just add the result of every 1x1x192 * 1x1x192
kerner result? So I'd get 28x28x1
matrix?
convnet
convnet
asked Sep 22 at 14:20
Mihkel L.
1233
1233
add a comment |Â
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).
I don't use torch, but keras, but the principle applies I think.
When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.
The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!
So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.
And that's the shrink suggested by Ng.
Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.
Edit: Simple example
I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.
import numpy as np
M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])
C = np.ones(shape=(2, 2))
c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
I'm trying to achive this with matrix multiplication. And there is noway I can get to28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what'sg
?
â Mihkel L.
Sep 23 at 7:46
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
best thing would be to show PyTorch code that compilesW x W x K
matrix intoW x W
matrix :)
â Mihkel L.
Sep 23 at 18:40
 |Â
show 2 more comments
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).
I don't use torch, but keras, but the principle applies I think.
When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.
The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!
So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.
And that's the shrink suggested by Ng.
Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.
Edit: Simple example
I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.
import numpy as np
M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])
C = np.ones(shape=(2, 2))
c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
I'm trying to achive this with matrix multiplication. And there is noway I can get to28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what'sg
?
â Mihkel L.
Sep 23 at 7:46
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
best thing would be to show PyTorch code that compilesW x W x K
matrix intoW x W
matrix :)
â Mihkel L.
Sep 23 at 18:40
 |Â
show 2 more comments
up vote
3
down vote
accepted
Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).
I don't use torch, but keras, but the principle applies I think.
When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.
The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!
So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.
And that's the shrink suggested by Ng.
Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.
Edit: Simple example
I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.
import numpy as np
M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])
C = np.ones(shape=(2, 2))
c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
I'm trying to achive this with matrix multiplication. And there is noway I can get to28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what'sg
?
â Mihkel L.
Sep 23 at 7:46
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
best thing would be to show PyTorch code that compilesW x W x K
matrix intoW x W
matrix :)
â Mihkel L.
Sep 23 at 18:40
 |Â
show 2 more comments
up vote
3
down vote
accepted
up vote
3
down vote
accepted
Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).
I don't use torch, but keras, but the principle applies I think.
When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.
The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!
So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.
And that's the shrink suggested by Ng.
Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.
Edit: Simple example
I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.
import numpy as np
M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])
C = np.ones(shape=(2, 2))
c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)
Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).
I don't use torch, but keras, but the principle applies I think.
When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.
The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!
So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.
And that's the shrink suggested by Ng.
Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.
Edit: Simple example
I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.
import numpy as np
M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])
C = np.ones(shape=(2, 2))
c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)
edited Sep 23 at 22:42
answered Sep 22 at 15:34
Francesco Pegoraro
51217
51217
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
I'm trying to achive this with matrix multiplication. And there is noway I can get to28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what'sg
?
â Mihkel L.
Sep 23 at 7:46
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
best thing would be to show PyTorch code that compilesW x W x K
matrix intoW x W
matrix :)
â Mihkel L.
Sep 23 at 18:40
 |Â
show 2 more comments
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
I'm trying to achive this with matrix multiplication. And there is noway I can get to28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what'sg
?
â Mihkel L.
Sep 23 at 7:46
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
best thing would be to show PyTorch code that compilesW x W x K
matrix intoW x W
matrix :)
â Mihkel L.
Sep 23 at 18:40
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
Yep! The catch here is that 2D filters isn't exactly 2D... And that confuses people
â Aditya
Sep 23 at 3:36
I'm trying to achive this with matrix multiplication. And there is noway I can get to
28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what's g
?â Mihkel L.
Sep 23 at 7:46
I'm trying to achive this with matrix multiplication. And there is noway I can get to
28x28xK
what's the calculation that shrinks it? If you look at @André answer, then what's g
?â Mihkel L.
Sep 23 at 7:46
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
You get the shrinking because the result of convolution between overlapping regions between a tensor of depth k and a filter of depth k is a matrix with depth 1! For each region you get a scalar, and all together they form a matrix. So you shrink from k to 1! To obtain a result of depth X, you just stack the result of X filters.
â Francesco Pegoraro
Sep 23 at 14:07
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
@Francesco Pegoraro Show me some real calculation. I have matrix 28x28x192. When I multiply it with 1x1x192 then i get 28x28x192. No Scalar. Show me how I get scalar please.
â Mihkel L.
Sep 23 at 18:30
best thing would be to show PyTorch code that compiles
W x W x K
matrix into W x W
matrix :)â Mihkel L.
Sep 23 at 18:40
best thing would be to show PyTorch code that compiles
W x W x K
matrix into W x W
matrix :)â Mihkel L.
Sep 23 at 18:40
 |Â
show 2 more comments
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f38643%2f1x1-convolution-how-does-the-math-work%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password